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Introduction 

Drawing  on  statistical  techniques  for  studying  disordered  systems  like  spin  glasses, 
physicists  have  made  remarkable  contributions  to  the  field  of  neural  networks  in  the  last 
decade.  Nevertheless,  hardware  implementations  of  theoretical  models  based  on  spin  glass 
ideas  remain  few.  An  important  reason  is  that  the  stochastic,  serial  updating  rules  of  these 
models  place  costly  demands  on  circuit  size  and  speed.  The  scarcity  of  implementations  is 
particularly  striking  because  neural  networks  must  move  off  general-purpose  computers 
and  onto  specialized  hardware  to  achieve  useful,  real-time  applicability. 

Rather  than  trying  to  force  theoretical  models  onto  hardware,  an  alternate  approach  is  to 
ask  what  features  are  readily  implementable  in  hardware  and  to  design  neural  networks 
accordingly.  The  research  carried  out  under  this  grant  has  explored  the  dynamics  of 
networks  of  analog  neurons — characterized  by  a  nonlinear  input-output  transfer  function 
with  finite  slope  or  gain — updated  deterministically ,  either  in  continuous  time  or  in  discrete 
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time  with  parallel  updating.  This  report  summarizes  the  most  important  results  on  the 
stability  of  these  networks  and  on  their  application  to  associative  memory  and  visual 
processing. 


Architecture  of  analog  neural  networks 

Two  network  architectures,  standard  and  competitive  analog  networks,  have  been 
studied  in  detail.  The  networks  are  depicted  schematically  in  Fig.  1. 


(a)  (b)  (c) 


Fig.  1.  Architecture  of  (a)  standard  and  (b)  competitive  analog  networks.  Circles  denote 
neurons,  filled  squares  denote  interconnections,  ellipses  in  (b)  denote  competitive  interaction,  (c) 
Analog  neuron  input-output  transfer  function. 


Standard  analog  networks  are  described  by  either  the  continuous-time  equations 


dXjjt) 

dt 


=  -xM  +  F 


N 


U=i 


i  = 


(1) 


or  by  the  discrete-time,  parallel-update  equations 
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Xi(t  + 1)  =  F 


N 

X  Jyxi® 


\  J=l 


i  =  1 


(2) 


where  jq(r)  is  the  output  of  neuron  i  at  time  t,  F  ( z )  is  a  nonlinear  input-output  transfer 
function,  and  Jtj  is  a  symmetric  interconnection  matrix.  These  networks  are  useful  for 
applications  involving  binary  data — for  example,  black  and  white  image  pixels. 

In  competitive  analog  networks,  neurons  compete  with  each  other  in  localized  clusters 
containing  Q  neurons  each.  The  networks  are  described  by  either  the  continuous-time 
equations 


dXjait) 

dt 


=  -xM  +  F 


I  N 

I 

;= 1  b= 1 


(3) 


or  by  the  discrete-time,  parallel-update  equations 


xia(t  +  l)  =  F 


i 

\  ;=1  b= l 


(4) 


where  xia(t)  is  the  output  of  neuron  a  in  cluster  i  at  time  t,  F  (z)  is  a  nonlinear  input- 
output  transfer  function,  and  is  a  symmetric  interconnection  matrix.  The  quantity  5,  (7) 
enforces  competition  in  cluster  i  by  making  the  outputs  of  all  neurons  in  the  cluster  sum  to 
a  constant  at  all  times.  The  winner-take-all  function,  in  which  only  one  neuron  having  the 
largest  input  fires  in  each  cluster,  is  an  example  of  competitive  behavior  implementable  with 
this  mechanism.  These  networks  are  useful  for  applications  involving  multiclass 
data — for  example,  image  pixels  of  several  different  colors. 


Stability  results 


Using  a  Liapunov  function  approach,  we  have  studied  the  stability  of  standard  and 
competitive  analog  networks.  Stability  is  important  in  designing  networks  that  compute  by 
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relaxing  to  fixed-point  attractors,  for  which  oscillation  is  undesirable.  Our  most  important 
result  is  that  analog  neurons  can  be  updated  in  parallel  without  oscillation,  a  crucial 
advantage  over  two-state  neurons  for  real-time  processing. 

With  continuous-time  updating,  both  standard  and  competitive  analog  networks  always 
converge  to  fixed-point  attractors.  With  discrete-time  parallel  updating,  both  network  types 
converge  only  to  fixed  points  when  the  slopes  of  the  transfer  functions  are  sufficiently 
small  but  can  oscillate  for  large  slopes.  The  condition  guaranteeing  convergence  is 
1/P  >  -Amm.  For  standard  networks,  p  is  the  steepest  slope  of  the  transfer  function,  and 
is  the  minimum  eigenvalue  of  J-tj.  For  competitive  networks,  p  is  a  measure  of  the 
steepness  of  all  Q  transfer  functions  in  a  cluster,  and  Amin  is  the  minimum  eigenvalue  of 

jab 

Jij  ■ 

Analog  associative  memories 

We  have  studied  the  attractors  of  standard  and  competitive  analog  networks  configured 
as  associative  memories  that  store  patterns  of  neuron  activity.  Our  most  important  results 
a~e  (i)  analytical  phase  diagram:,  describing  the  different  attractor  types  as  a  function  of  the 
transfer  function  slope  P  and  the  ratio  a  of  patterns  to  neurons,  and  (ii)  analytical  results 
showing  that  the  number  of  spurious  attractors  decreases  as  the  transfer  function  slope  is 
reduced.  The  phase  diagrams  provide  quantitative  guidelines  for  designing  and  operating 
associative  memories  by  indicating  regions  of  parameter  space  where  memory  recall  is 
possible  as  well  as  regions  where  spurious  fixed  points  or  oscillatory  attractors  exist.  The 
results  on  spurious  attractors  have  implications  for  improving  performance  in  analog 
computers  that  solve  complex  optimization  problems. 

A  typical  phase  diagram  is  shown  in  Fig.  2(a)  for  standard  analog  associative  memories 
with  discrete-time,  parallel  updating.  The  diagram  has  four  regions  called  recall, 
oscillatory,  spin  glass,  and  paramagnetic.  In  the  recall  region,  the  networks  operate 
reliably  as  associative  memories,  with  fixed  points  corresponding  to  stored  patterns;  in  the 
oscillation  region,  the  stability  criterion  discussed  above  is  violated;  in  the  spin  glass 
region,  no  recall  fixed  pomts  exist,  but  there  are  numerous  spurious  fixed  points;  and  in  the 
paramagnetic  region,  there  is  a  single  attractor  with  all  neuron  outputs  zero. 
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Fig.  2.  (a)  Phase  diagram  showing  recall,  oscillatory,  spin  glass,  and  paramagnetic  regions,  (b) 

Exponent  g(a,p).  Number  of  spurious  fixed  points  varies  as  eNg . 

Results  on  spurious  attractors  in  standard  analog  associative  memories  are  shown  in 
Fig.  2(b).  We  find  that  the  number  of  spurious  attractors  varies  with  network  size  N  as 
eNg.  The  exponent  g(a,{ 3)  depends  on  the  transfer  function  slope  (3  and  on  the  ratio  a 
of  patterns  to  neurons.  As  shown  in  the  figure,  the  exponent  decreases  as  the  slope 
decreases.  This  implies  that  the  number  of  spurious  attractors  is  reduced  and  the  network 
performance  improved  as  the  slope  is  lowered. 

Analog  neural  networks  for  visual  processing 

We  have  also  investigated  two-dimensional  analog  networks  for  translation-invariant 
feature  detection  in  visual  scenes.  Our  main  result  is  that  competitive  analog  networks  are 
extremely  well-suited  for  visual  feature  detection,  performing  difficult  tasks  with  simple, 
sparse  wiring  configurations. 
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Fig.  3.  Architecture  of  a  two-dimensional  competitive  feature  detector. 

Figure  3  depicts  the  architecture  of  a  competitive  feature  detector.  Each  neuron  in  a 
competitive  cluster  represents  a  fragment  of  a  feature  to  be  detected.  Neurons  receive 
inputs  both  from  a  localized  region  of  a  pixel  array  and  from  neurons  in  neighboring 
clusters.  The  interconnections  between  clusters  are  designed  to  reconstruct  complete 
features  from  their  constituent  fragments.  These  networks  can  successfully  detect  features 
in  noisy  images  with  translation  invariance  on  the  length  scale  of  the  feature  fragments. 

Figure  4  shows  the  initial  and  final  pixel  array  of  a  network  that  recognizes  two 
features,  a  star  and  a  triangle.  Each  7x7  pixel  block  feeds  into  a  single  cluster  of  9 
competing  neurons;  the  entire  network  needs  only  6  x  6  =  36  clusters,  36  x  9  =  324 
neurons,  and  36  x  20  =  720  interconnections  to  process  an  array  of  36  x  49  =  1764  pixels. 
The  savings  in  wiring — the  interconnections  scale  linearly  with  pixels — comes  about 
because  interconnections  transmit  information  about  entire  features,  rather  than  individual 
pixels. 
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Fig.  4.  (a)  Initial  and  (b)  final  pixel  array  of  a  competitive  feature  detector. 
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